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PREFACE 


This  report  summarizes  the  progress  and  findings  to  date  of 
work  conducted  to  develop  a  training  systems  design/"lessons 
learned"  database-  The  work  described  is  part  of  a  larger  program 
concerned  with  the  development  of  principles  and  guidelines  for  the 
design,  development,  evaluation,  and  operation  of  total  aircrew 
training  systems,  which  is  being  accomplished  by  the  University  cf 
Dayton  Research  Institute  (UDRI)  under  Contract  F33615-90-C-0005 
with  Aircrew  Training  Research  Division  of  the  Armstrong 
Laboratory’s  Human  Resources  Directorate  (AL/HRA) .  The  contract 
monitor  was  Capt  Claire  Fitzpatrick.  The  work  was  conducted  under 
Work  Unit  1123-03-85,  Flying  Training  Research  Support. 

The  authors  wish  to  express  their  appreciation  to  all  of  the 
Government  civilian  and  military  personnel  who  gave  freely  of  their 
time  and  provided  the  information  which  made  this  review  possible. 
Special  thanks  are  also  due  to  Ms.  Marge  Keslin  of  UDRI,  who 
provided  invaluable  support  throughout  the  preparation  and 
publication  of  this  report. 
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CONTRACTOR-SUPPORTED  AIRCREW  TRAINING  SYSTEMS: 

ISSUES  AND  LESSONS  LEARNED 

SUMMARY 

The  U.S.  Air  Force's  experience  with  respect  to  the  design  of 
total  aircrew  training  systems  (ATS)  is  quite  limited.  To  date, 
most  of  the  major  Air  Force  ATS  programs  have  not  been  acquired 
and/or  completely  implemented.  Thus,  there  is  a  pressing  need  to 
develop  and  disseminate  empirically  validated  principles,  criteria, 
and/or  "lessons  learned"  to  training  system  acquisition  offices  and 
operational  training  organizations  to  ensure  a  cost-effective 
approach  in  the  design,  development,  and  use  of  total  aircrew 
training  systems.  In  response  to  this  need,  AL/HRA  initiated  a 
program  to  perform  the  analyses  and  research  required  to  develop  a 
total  training  system  design  database  to  support  the  preparation  of 
user-oriented  guidelines  and  to  serve  as  a  research  and 
development  resource. 

A  comprehensive  review  was  made  of  several  Air  Force  ATS 
programs  (e.g.,  KC-10,  C-5,  and  C-130) .  The  primary  purpose  of 
this  review  was  to  identify  the  major  issues  and  lessons  learned 
during  the  design,  development,  evaluation,  and  operation  of  these 
systems.  In  addition,  data  were  also  obtained  from  selected  Navy 
and  allied  forces  aircrew  training  systems  tc  serve  as  a  cross¬ 
check  on  the  generality  of  the  Air  Force  findings.  The  major 
findings  of  the  ATS  review  are  summarized,  and  the  key  issues  and 
lessons  learned  are  identified  and  discussed. 

INTRODUCTION 

This  report  summarizes  the  progress  and  findings  to  date  of 
work  conducted  to  develop  an  aircrew  training  system  design/ 
"lessons  learned"  database.  The  work  described  is  part  of  a  larger 
program  concerned  with  the  development  of  principles  and  guidelines 
for  the  design,  development,  evaluation,  and  operation  of  total 
aircrew  training  systems. 
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Background 


The  current  trend  within  the  U.S.  Air  Force  is  to  design 
aircrew  training  programs  as  total  integrated  systems  rather  than  as 
collections  of  courses  or  blocks  of  instruction.  This  trend  has 
been  coupled  with  a  concurrent  shift  to  contracting  out  the  design, 
delivery,  and  support  of  aircrew  training.  These  changes  have 
introduced  a  new  set  of  technical  and  management  issues  which 
impact  the  design,  development,  evaluation,  and  operation  of 
aircrew  training  programs.  The  Aircrew  Training  Research  Division 
(AL/HRA)  is  currently  conducting  research  and  development  (R&D)  to 
address  several  of  these  issues  in  order  to  provide  principles, 
procedures,  and  user-oriented  guidelines  to  support  Air  Force 
acquisition  and  operational  training  agencies. 

Many  current  aircrew  training  programs  have  been  characterized 
as  being  largely  lock-step,  labor-intensive,  and  inefficient. 
Frequently,  these  programs  are  comprised  of  phases  and  blocks  of 
instruction  which  have  been  independently  designed  and  developed  in 
relative  isolation  from  the  "system"  as  a  whole.  As  a  consequence, 
there  are  frequent  gaps  or  unnecessary  redundancies  across  blocks 
of  training.  It  is  often  the  case  that  the  aircraft  itself,  which 
is  usually  the  most  costly  resource,  is  used  to  compensate  for 
instructional  deficiencies  that  could  have  been  corrected  in  much 
less  expensive  ways.  In  addition,  because  of  the  lack  of 
integration  across  components  and  subsystems  of  the  total  system, 
current  programs  are  often  relatively  inflexible  and/or  inefficient 
in  adapting  to  changes  such  as  those  imposed  by  modifications  in 
operational  requirements,  variations  in  student  flow,  and  the 
introduction  of  new  training  technologies.  For  instance,  often 
when  a  new  technology  (e.g.,  a  high-capability  flight  simulator)  is 
introduced  into  an  existing  program,  it  is  treated  as  an  add-on 
cost  for  local  facilities  and  operations  and  maintenance  (O&M) 
rather  than  as  an  opportunity  to  improve  the  overall  cost- 
effectiveness  of  the  total  training  system. 

As  a  result  of  such  deficiencies,  there  is  a  growing 
recognition  within  the  Air  Force,  as  well  as  the  other  military 
services,  of  the  need  to  design,  develop,  evaluate,  and  operate 
aircrew  training  programs  as  total  systems  rather  than  as  relative¬ 
ly  disjointed  sets  of  components  and/or  phases  of  training. 
Currently,  the  Air  Force  is  involved  in  the  development/operation 
of  a  number  of  aircrew  training  programs  from  a  total  training 
system  point  of  view.  Virtually  all  of  these  programs  are  being 
developed  and  operated  with  some  measure  of  contractor  support. 
For  example,  one  of  the  major  new  initiatives  is  the  C-130  Aircrew 
Training  System  (ATS) ,  which  is  currently  being  developed  for  the 
Military  Airlift  Command  (MAC)  at  Little  Rock  AFB,  Arkansas,  under 
a  contract  with  the  CAE-Link  Corporation.  In  addition  to  the  C-130 
ATS,  each  of  the  Air  Force  using  commands  is  also  planning, 
developing,  and/or  operating  a  number  of  other  contractor-supported 
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aircrew  training  programs  such  as  the  E-3A,  F-15,  and  F-16 
(Tactical  Air  Command) ;  KC-10  and  B-52/KC-135  (Strategic  Air 
Command);  C-5,  C-141,  C-17,  and  Special  Operations  Forces  (SOF) 
(MAC) ;  and  the  TTTS  (Tanker-Transport  Training  System)  of  the  Air 
Training  Command. 

Most  of  the  Air  Force's  major  ATS  programs  are  only  in  the 
early  stages  of  the  system  life  cycle.  As  a  consequence,  there  is 
relatively  little  empirical  and/or  experiential  information  to 
provide  guidance  for  the  design,  development,  evaluation,  and 
operation  of  new  or  proposed  systems.  In  fact,  in  most  of  the 
systems  reviewed  to  date  the  program  managers  began  at  ground  zero 
and  had  to  "learn  by  doing,"  except  for  occasional  consultation 
with  personnel  in  other  programs  who  were  somewhat  farther  along 
than  they  were.  To  promote  communication  among  the  various 
programs  and  agencies  involved  in  the  development  and  operation  of 
total  aircrew  training  systems,  AL/HRA  and  Training  System  SPO  of 
the  Aeronautical  Systems  Division  (ASD/YW)  jointly  established  the 
Training  System  Training  Effectiveness  Working  Group  (TEWG) .  This 
TEWG  has  proved  to  be  a  very  useful  forum  for  facilitating 
inter-program  communication  and  information  exchange.  However,  to 
be  solely  dependent  on  occasional  personal  contacts  and/or 
undocumented  corporate  memory  for  guidance  is  a  less-than-optimum 
procedure  for  use  in  the  development  of  major  training  programs. 
Thus,  there  is  a  critical  need  to  systematically  collect  and 
document  experientially  based  "lessons  learned"  and  disseminate 
them  to  training  system  acquisition  offices  and  operational 
training  organizations  to  ensure  a  more  cost-effective  approach  in 
the  design,  development,  and  evaluation  of  both  ongoing  and  future 
aircrew  training  systems.  Because  of  the  number  of  aircrew 
training  systems  at  various  stages  of  development  at  the  present 
time,  a  window  of  opportunity  is  available  for  collecting  lessons 
learned  during  various  phases  of  the  training  system  life  cycle. 

Objectives 

The  general  objective  of  this  program  is  an  aircrew  training 
system  design/lessons  learned  database  for  use  as  an  R&D  resource 
and  as  a  source  of  information  for  the  development  of  user-oriented 
guidelines  for  the  cost-effective  design,  development,  and 
evaluation  of  total  integrated  aircrew  training  systems.  The 
primary  objectives  of  this  specific  effort  were  (a)  to  collect 
lessons  learned  by  Air  Force  and  contractor  personnel  in  selected 
Air  Force  contractor-supported  aircrew  training  programs,  and  (b) 
to  identify  and  document  key  issues/problem  areas  which  could  serve 
as  a  structure  for  follow-on  data  collection,  as  well  as  provide  a 
focus  for  the  development  of  a  high-payoff  R&D  plan. 

An  ancillary  objective  was  to  solicit  issues/lessons  learned 
in  contractor-supported  aircrew  instruction  from  the  various  allied 
military  agencies  represented  by  the  members  of  the  Training 
Technology  Panel  (UTP-2)  of  The  Technology  Cooperation  Program 
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(TTCP) .  (The  TTCP  membership  includes  representatives  from  the 
military  services  of  the  United  Kingdom,  Canada,  Australia,  New 
Zealand,  and  the  United  States.)  The  TTCP  review  was  requested  by 
Dr.  Wayne  Waag  (AL/HRA) ,  the  Chairman  of  UTP-2 ,  following  a  strong 
show  of  interest  by  members  of  the  panel  regarding  a  point  paper  on 
"Issues  in  Contracted  Instruction"  prepared  by  Dr.  Robert  Nullmeyer 
of  AL/HRA  (see  Appendix  A) . 

Purpose  of  the  Present  Report 

The  primary  purpose  of  the  present  report  is  to  summarize  the 
activities  and  findings  of  this  effort  to  date  and  to  describe  the 
goals  and  objectives  for  the  future.  The  report  is  divided  into 
six  major  sections.  Introduction  provides  the  rationale  and 
describes  the  objectives  of  this  effort.  Method  summarizes  the 
general  approach  for  the  entire  multi-year  program,  as  well  as  the 
specific  approach  used  in  this  effort.  It  also  lists  the  training 
systems  reviewed  during  the  period  covered  by  this  report  and 
identifies  some  of  the  major  information  sources.  Findings 
summarize  the  major  lessons  learned  and  issues  identified. 
Discussion  considers  some  of  the  implications  of  the  findings  to 
date  and  discusses  the  plans  for  follow-on  actions.  Bibliography 
lists  information  sources  for  this  research.  Finally,  Appendix  A 
and  Appendix  B  contain  documentation  that  augments  the  information 
contained  in  the  main  body  of  the  report. 


METHOD 

General  Approach 

The  approach  for  the  overall  program  consists  of  three 
overlapping  phases. 

Phase  1.  The  first  phase  is  a  continuing  cross-sectional  and 
longitudinal  data  collection  effort.  It  includes  a  series  of 
visits  to  sites  where  total  aircrew  training  systems  are  being 
planned,  developed,  and/or  operated.  Major  training  system  issues 
and  lessons  learned  are  solicited  from  both  contractor  and  military 
personnel  involved  in  all  phases  of  the  training  system  life  cycle. 
Much  of  the  information  being  developed  is  based  on  personal 
interviews  as  well  as  the  authors'  personal  participation  in 
selected  programs.  During  the  initial  contacts,  specific  lessons 
learned  and  issues  are  identified  through  somewhat  open-ended 
discussions.  Where  feasible,  an  effort  is  made  to  obtain 
additional  information  concerning  what  problems  are/were  being 
experienced  and  what  approaches,  techniques,  and  procedures  were 
developed  to  resolve  them  that  might  be  of  value  to  others. 
Several  potentially  profitable  areas  and  issues  have  been 
identified;  these  are  being  monitored  through  periodic  visits, 
telephone  communications,  and  structured  questionnaires. 
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Phase  2.  During  the  second  phase  of  this  program,  several 
generic  ATS  issues/functions  are  being  selected  for  further 
in-depth  review  and  analysis.  Among  the  specific  areas  already 
identified  for  more  detailed  study  are  (a)  test  and  evaluation, 
(b)  aircrew  performance  measurement  methodology,  (c)  techniques  for 
management  of  the  courseware  development  process,  (d)  training 
management  system  design  (including  scheduling  and  resource 
allocation) ,  and  (e)  the  use  of  flight  simulators  for  combat- 
oriented  aircrew  training. 

Phase  3.  During  Phase  3,  the  information  concerning  the  major 
issues  and  lessons  learned  will  be  documented  in  a  series  of  user- 
oriented  reports  and  guidelines  for  dissemination  to  procurement 
personnel,  training  system  designers,  training  system  acquisition 
agencies,  training  managers,  and  R&D  personnel.  The  guidelines 
should  effect  a  significant  improvement  in  the  effectiveness  and 
efficiency  of  system  design,  development,  evaluation,  and 
utilization  processes.  Tn  addition,  the  lessons  learned  database 
should  be  a  prime  source  of  guidance  for  identifying,  organizing, 
and  prioritizing  significant  training  system  R&D  issues. 

Specific  Approach 

The  primary  emphasis  during  the  initial  phase  of  the  program 
was  to  establish  contact  with  a  number  of  major  Air  Force  aircrew 
training  programs  that  could  serve  as  continuing  sources  of 
information  throughout  this  effort.  In  addition,  an  attempt  was 
made  to  identify  other  aircrew  training  programs  in  the  Air  Force, 
as  well  as  other  military  agencies  that  might  merit  more  thorough 
review  and  analysis  in  the  future.  Visits  were  made  to  a  number  of 
contractor  and  military  facilities  to  interview  key  management  and 
technical  personnel  in  order  to  gain  a  better  understanding  of  the 
program (s)  in  which  these  personnel  were  involved  and  to  solicit 
their  opinions  concerning  important  issues  and  lessons  learned  to 
date.  In  most  cases,  follow-on  visits  and/or  telephone  interviews 
were  conducted  to  clarify  particular  points  and  to  obtain  addi¬ 
tional  information.  In  addition  to  the  personal  interviews, 
considerable  time  was  spent  reviewing  program  documentation  and 
other  relevant  published  information.  Another  source  of  data  for 
the  authors  was  derived  from  their  own  participation  in  various 
phases  of  the  C-130  MATS  (Model  Aircrew  Training  System) ,  C-130 
ATS,  B-52/KC-13 5  Combat  Crew  Training  School  (CCTS)  Modernization, 
and  the  SOF  ATS  programs.  The  primary  information  sources  and 
systems  covered  during  this  first  phase  are  shown  in  Table  1,  and 
a  listing  of  some  of  the  major  U.S.  Air  Force  and  Navy  contractor- 
supported  systems  appears  in  Table  2. 

Most  of  the  initial  interviews  with  key  program  personnel  were 
relatively  unstructured  and  open-ended.  However,  later  interviews 
tended  to  become  more  focused  as  a  number  of  major  issues  which 
appeared  to  be  common  across  several  systems  began  to  emerge. 
These  issues  were  also  used  to  structure  a  questionnaire  that  was 
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Table  1 


Major  U.8.  Sources  for  Issues/"Lessons  Learned"  Data 


Organization 

System  Is) 

CAE-Link 

C-130,  C-5 

CAE-Link 

C-130 

MAC 

C-5 

MDTS 

KC-10 

MDCBTS 

C-17 ,  TTTS 

SA2 

KC-10,  B-52/KC-135 

AL/HRA 

B-52/KC-135 

USAC 

E-4 

MAC 

C-130 

AL/HRA 

F-5 

MDCBTS 

T-4  5 

MAC 

C-5 

TAC 

E-3A,  F-15/F-16 

MAC 

C-130 

Arizona  ANG 

F-16/A-7 

MDCBTS 

T-4  5 

USAC 

C-5 

NTSC 

E-6A 

SAC 

KC-10 

TAC 

E-3A 

MAC 

C-5 

Navy 

F-18 

Table  2. 

Some  Major  U.S. 
Training  Systems 

Contractor-Supported  Aircrew 

Sponsor 

Aircraft 

Prime  contractor 

USAF  (MAC) 

C-130 

CAE-Link 

USAF  (MAC) 

C-5 

Flight  Safety  International 

USAF  (MAC) 

C-17 

McDonnell  Douglas 

USAF  (SAC) 

KC-10 

McDonnell  Douglas 

USAF  (TAC) 

E— 3 

Boeing  Simulation 

USN 

E-6A 

McDonnell  Douglas 

USN 

T-47 

Cessna 

USN/USMC 

EA-6B 

Simuf lite 

USN/ U SMC 

T-4  5 

McDonnell  Douglas 
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used  to  survey  UTP-2  members  regarding  contractor-supported 
training.  The  TTCP  questionnaire  and  supporting  materials  are 
contained  in  Appendix  A. 


FINDINGS 

The  primary  data  collected  during  this  phase  were  lessons 
learned  by  key  personnel  in  the  programs  selected  for  review. 
Because  the  initial  sample  of  interviewees  consisted  mainly  of 
contractor  and  military  program  and/or  site  managers,  most  of  the 
data  relate  to  procurement  and  management  issues.  In  addition, 
most  of  the  major  systems  reviewed  were  still  under  development  or 
in  the  early  stages  of  implementation  at  the  time  the  data  were 
obtained.  Thus,  most  of  the  systems  did  not  have  sufficient 
operational  run  time  to  assess  or  validate  the  longer-term  impacts 
of  many  of  the  early  technical  and  management  decisions  taken.  In 
addition,  although  there  is  a  fair  amount  of  consistency  in  the 
lessons  learned  identified  by  the  personnel  interviewed  to  date, 
there  are  also  a  few  apparent  inconsistencies.  In  general,  these 
result,  at  least  in  part,  from  obvious  differences  between 
Government  and  contractor  perspectives;  differences  in  the  kinds  of 
systems  and  the  levels  of  contractor  or  Government  involvement;  and 
differences  in  the  operational,  training,  and  management 
philosophies  among  the  various  military  agencies. 

Some  of  the  representative  lessons  learned  data  obtained  from 
personnel  involved  in  the  0130,  05,  and  KC-10  ATS  programs  are 
summarized  in  this  section  for  illustrative  purposes.  Where 
appropriate,  the  data  are  also  listed  as  being  derived  from  a 
contractor  or  Air  Force  source.  The  0130,  05,  and  KC-10,  0141, 
017 ,  and  the  SOF  ATS  will  be  among  the  primary  sources  continually 
monitored  for  follow-on  data  collection.  A  separate  subsection 
documents  lessons  learned  by  AL/HRA  in  working  with  more  than  one 
program. 

An  index  of  the  issues/lessons  learned  for  the  core  systems  is 
contained  in  Appendix  C.  Each  lesson  learned  is  labeled  with  a 
provisional  database  field  for  classification  and  filing  purposes. 
(The  current  field  structure  may  be  modified  in  the  future  as 
additional  data  become  available.) 

C-130  ATS 


C-130  ATS  Description 

The  C-130  ATS  is  an  integrated,  contractor-supported  training 
system  being  developed  to  provide  ground-based  training  for  C-130 
aircrews.  The  C-130  ATS  contract  includes  28  courses  for  the 
Department  of  Defense  (DoD)  formal  school  at  Little  Rock  AFB,  and 
all  C-130E-  and  H-model  continuation  training.  The  system  includes 
the  optimized  use  of  existing  training  assets,  including  10  C-130 
Weapon  System  Trainers  (WSTs) ;  two  cockpit  procedures  trainers 
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(CPTs) ;  and  several  part-task  trainers  (PTTs) ,  which  are  to  be 
furnished  "as  is"  to  the  contractor  by  the  Government.  It  also 
includes  all  maintenance  and  logistics  support  for  the  WSTs  and 
other  PTTs  within  the  program.  It  includes  total  system  management 
of  all  ground-based  training  using  computerized  management  tools, 
all  scheduling,  and  all  training  scenarios  for  the  flying  environ¬ 
ment.  It  also  includes  a  proficiency-based  training  continuum  which 
begins  with  entry  intc  the  formal  school  and  ends  with  either 
transfer  out  of  the  weapon  system  or  retirement. 

Under  the  C-130  ATS  concept,  the  contractor  is  responsible  for 
the  entire  Instructional  Systems  Development  (ISD)  process  from 
beginning  to  end,  including  formative,  summative,  and  operational 
evaluations.  The  contractor  is  responsible  for  the  development  and 
production  of  all  courseware,  all  ground  instruction,  all  hardware 
modifications,  and  any  new  software  development.  The  contractor  is 
also  responsible  for  the  total  operation,  maintenance,  and  support 
of  the  ground-based  training  system;  all  student  management; 
administration;  configuration  management;  and  quality  assurance. 

C-130  ATS  Issues/Lessons  Learned 

Air  Force  Issues/Lessons  Learned 

Contractor  Education :  Contractors  must  go  through  a  relatively  long 
familiarization/educational  process  before  they  know  enough  about 
a  given  system  to  conduct  an  effective  and  efficient  ISD  process. 

Computer-Based  Training  (CBT)  Costing :  There  is  a  general  tendency  to 
greatly  underestimate  the  time  and  cost  required  to  develop  good 
CBT  lessons.  (At  the  time  of  this  review,  one  of  the  contractor's 
CBT  development  personnel  noted  that  it  was  taking  about  400  hours 
to  develop  a  1-hour  CBT  lesson.) 

Student  Evaluation/ Guarantee :  There  should  be  a  contractual 
requirement  to  ensure  that  all  check  ride  evaluations  be  tied  back 
to  training  objectives.  In  fact,  documentation  of  the  correlation 
between  check  ride  performance  and  contractor-identified  objectives 
should  be  made  a  contract  deliverable.  It  is  acknowledged, 
however,  that  it  is  not  always  easy  to  identify  a  direct  link 
between  all  relevant  enabling  objectives  and  the  particular  sample 
of  aircrew  behaviors  evaluated  during  the  check  ride,  for  a  single 
check  ride  is  simply  the  top  of  the  student  evaluation  pyramid. 
Within  a  single  course,  the  contractor  is  responsible  for  teaching 
hundreds  of  objectives,  many  of  which  are  not  tested  in  the 
aircraft  (e.g.,  ditching,  serious  malfunctions).  Therefore,  it  is 
necessary  that  the  means  and  the  results  of  all  tests/checks  be 
made  visible  to  the  Government  sponsor. 

Course  Evaluation  Data ;  The  contract  should  stipulate  that  the 
Government  be  provided  with  every  writeup  generated  by  individual 
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tryouts  during  the  course  development  process.  This  should  help 
ensure  that  when  individual  tryouts  have  been  completed,  all  major 
courseware  technical  issues  have  been  resolved. 

On-Siie  Development :  It  is  highly  advantageous  to  have  the 
contractor  develop  the  training  program  on  the  site  where  it  will 
be  implemented.  This  facilitates  day-to-day  communication  with  the 
user,  which  expedites  information  gathering  and  tends  to  promote 
better  design  decisions.  Perhaps  this  requirement  may  be  relaxed 
somewhat  for  a  "brand  new  system  like  the  C-17,  where  there  is  no 
available  operational  expertise." 

Management:  There  is  a  need  to  have  dedicated  Air  Force 
operational  specialists  on  site  during  the  development,  test,  and 
evaluation  of  an  ATS.  These  personnel  should  serve  as  the  con¬ 
tractor's  official  focal  points  for  subject-matter  expertise  in 
each  of  the  crew  positions  represented  in  the  system.  In  addition, 
they  can  support  the  acquisition  agency  by  providing  day-to-day 
oversight  of  contractor  progress  and  performance.  In  the  C-130 
ATS,  HQ  MAC  established  MACOS  OL  Q  (Military  Airlift  Combat 
Operational  Support  Operating  Location  Q)  at  Little  Rock  AFB  to 
provide  these  functions.  MACOS  OL  Q  consisted  of  11  persons, 
including  two  civilian  quality  assurance  specialists.  The  military 
personnel  were  all  rated  specialists  and  on  controlled  tours  until 
PMRT  (Program  Management  Responsibility  Transfer) . 

Course  Content :  Fearing  potential  legal  problems,  the  contractor 
was  hesitant  to  teach  useful  "techniques"  not  identified  in  the 
official  technical  orders.  To  ensure  that  appropriate  techniques 
would  not  be  excluded  from  the  new  program,  a  policy  was 
established  to  enable  the  contractor  to  submit  proposed  technique 
information  to  the  Air  Force  for  approval  prior  to  inclusion  in  the 
program. 

Configuration  Cuntroll System  Baseline :  Training  systems  are  much  more 
dynamic  than  most  operational  hardware/software  systems.  This 
creates  particular  problems  in  the  courseware  and  the  training 
management  system  areas  where  any  given  change  tends  to  ripple  to 
other  elements  of  the  system.  For  example,  a  change  in  the  content 
of  a  specific  lesson  imposed  by  changes  in  operational  equipment, 
procedures,  instructional  approach,  etc.  usually  will  affect  other 
elements  of  the  system  such  as  test  items,  student  orientation 
manuals,  and  course  summary  documents.  In  the  C-130  ATS,  we  have 
tailored  the  traditional  configuration  management  procedures 
(AFR  57-4,  Modification,  Approval,  and  Management)  to  ensure 
that  we  identify  the  impact  of  any  proposed  change  on  other 
parts  of  the  system  before  the  change  is  approved.  When  a 
change  is  implemented,  the  underlying  system  baseline  documenta¬ 
tion  is  also  updated  so  that  the  Air  Force  is  always  in  a 
position  to  permit  recompetition  and  reprocurement  when  necessary. 
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Contract /Proposal:  One  respondent  stated,  "Be  sure  to  put  the 
contractor's  proposal  on  contract!"  (Note:  This  lesson  learned  was 
identified  as  one  of  the  most  important  provided  by  this 
respondent.  It  was  a  reaction  to  the  tendency  for  some  contractor 
bidders  to  write  Impressive  proposals  which  are  ignored  after 
contract  award  because  the  proposals  were  not  contractually 
binding. ) 

Front-End  Analysis  (FEA)  Cost  and  Data:  The  C-130  ATS  program  evolved 
from  a  pre-contractual  front-end  analysis  and  preliminary  design 
study.  This  detailed  FEA  was  a  major  contributor  to  the  success  of 
the  program.  However,  a  major  lesson  learned  was  that  the  job/task 
analysis  data  should  be  delivered  on  magnetic  media  in  a  format 
that  is  compatible  with  the  user's  computer  capability.  Another 
lesson  learned  was  that  the  task  analysis  consumed  the  bulk  of  the 
contract  resources  for  the  FEA.  The  FEA  is  a  very  labor-intensive 
enterprise,  but  it  is  key  to  the  development  of  a  relevant  training 
program. 

Statement  of  Work  (SOW) :  The  FEA  served  as  the  baseline  for  the 
SOW.  The  SOW  should  permit  the  contractor  flexibility  in  his 
approach,  but  should  identify  the  constraints  within  which  he  must 
operate.  It  should  include  the  baseline  analysis  and  training 
requirements  as  part  of  the  package.  It  should  also  detail  site 
activation  requirements  and  the  requirement  to  develop  and  maintain 
a  recompetition  package,  as  well  as  concurrency  management  of  the 
hardware  and  software  within  the  system. 

Instructions  to  Offerers  (ITOs):  It  is  very  important  that  the 
instructions  to  offerers  conform  to  the  statement  of  work  and 
system  specification  in  order  to  facilitate  tracking  during  the 
proposal  evaluation  process. 

Source  Selection:  it  is  important  to  ensure  that  highly  competent 
instructional  technologists  and  training  experts  be  included  as 
members  of,  or  consultants  to,  the  source  selection  team  because  of 
the  difficulty  of  identifying  definitive  criteria  for  an  acceptable 
training  system  design. 

In  the  hardware/software  areas,  the  specification  of  "best 
commercial  practice"  provides  contractors  with  greater  flexibility 
as  to  how  they  will  satisfy  a  given  set  of  requirements,  but  it 
also  demands  a  greater  level  of  expertise  on  the  part  of  the  Air 
Force  reviewers  to  evaluate  the  feasibility  and  potential  risk  of 
a  proposed  technical  approach. 

Program  Cost  Evaluation:  Contractor  cost  evaluations  should  be 
based  solely  upon  the  proposed  ground  training  program.  The  SOW 
should  require  an  optimized  flying  hour  program  based  on  the 
contractor's  proposed  ground  training  program  and  evaluated  solely 
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on  technical  content.  This  is  a  feasible  approach  as  the  cost  of 
flying  hours  is  a  command  operations  and  maintenance  (O&M)  require¬ 
ment  and  the  final  decision  on  use  of  these  hours  falls  within  the 
using  command's  authority,  not  within  the  jurisdiction  of  the 
contractor  or  procuring  agency.  (Authors'  note:  This 
recommendation  may  not  be  applicable  for  ATSs  like  the  E-3A  and 
E-6A  where  the  contractor  is  responsible  for  the  O&M  of  training 
aircraft. ) 

Courseware  Costing'.  It  is  beneficial  to  include  a  requirement 
(and  the  funds)  for  timely  courseware  updates  at  no  additional 
charge  to  the  contract.  This  will  allow  changes  to  technical 
orders  (T.O.s),  flight  manuals,  etc.  (which  affect  the  training 
program)  to  be  accommodated  without  having  to  go  through  another 
costing  loop  and/or  contract  amendment  for  every  change. 

Student  Costing:  The  contract  should  include  a  provision  and  a 
source  of  funds  to  cover  the  additional  costs  that  may  be  incurred 
because  of  unprogrammed  surges  in  student  input  or  changes  in 
instructional  resources.  (Authors'  note:  These  contingencies  have 
been  covered  in  some  training  systems  by  factoring-in  the  cost  of 
an  estimated  change  factor  and/or  providing  a  "management  reserve" 
contingency  fund  to  cover  unanticipated  changes.) 

Task/ Objective  Data:  The  contract  should  be  absolutely  clear  about 
the  level  of  detail  to  which  tasks  and  objective  data  are  to  be 
developed.  These  details  will  result  in  satisfying  and  tracking 
the  training  requirements  of  the  guaranteed  student. 

Production  Tracking:  A  system  is  needed  to  track  the  status  of 
courseware  (as  well  as  hardware,  software,  etc.)  during 
development.  For  any  large  program,  it  is  necessary  to  automate 
the  process  in  order  to  adequately  monitor  the  hundreds  of  separate 
items  under  development.  In  the  C-130  ATS  program,  a  multi-step 
courseware  development  process  was  implemented.  Each  lesson 
development  plan  had  a  schedule  for  meeting  the  requirements  of 
each  step.  The  information  was  fed  into  a  computer  for  tracking 
purposes,  and  the  lesson  was  flagged  for  potential  management 
attention  if  it  was  ahead  or  behind  its  proposed  schedule  by  3 
days. 


System  Integration:  A  training  system  must  be  developed  as  a  total 
integrated  system.  Unfortunately,  because  of  the  lack  of 
experience  with  large-scale  training  systems  (as  opposed  to 
courses)  ,  it  is  too  easy  to  fractionate  the  development  effort 
among  interrelated  specialists  and  functions  in  the  drive  to  meet 
scheduled  milestones.  This  can  lead  to  problems  at  critical  system 
interfaces  and  produce  inefficient  and/or  ineffective  system 
performance. 
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Guaranteed  System :  It  is  necessary  to  exercise  program  management 
disciplines  such  as  quality  assurance,  quality  control,  and  test 
and  evaluation  to  ensure  that  the  overall  training  system  is 
meeting  all  of  its  specified  goals  and  functional  requirements 
throughout  the  system  life  cycle.  (Authors'  note:  During  the 
development  phase  of  the  C-130  ATS,  the  on-site  HQ  MAC  unit,  MACOS 
OL  Q,  performed  many  of  the  required  functions  for  the  Air  Force. 
Currently,  the  34  TATG  (Tactical  Airlift  Training  Group)  has 
established  a  special  unit  at  Little  Rock  AFB  to  continuously 
review  and  monitor  system  performance  for  the  duration  of  the 
system  life  cycle.) 

Contractor  Issues/Lessons  Learned 

Organization/Management'.  The  Test  and  Evaluation  function  should 
report  to  the  Program  Manager. 

Organization/Management/Development:  The  Test  and  Evaluation  process 
supports  system  integration. 

Test  and  Evaluation:  As  much  development  as  possible  should  be 
completed  during  formative  evaluation.  Small-group  tryouts  should 
employ  finished  courseware,  hardware/software,  and  media  and  should 
run  the  entire  course  from  beginning  to  end. 

Organization/Management:  The  Air  Force  and  the  contractor  should 
function  together  as  a  team  and  not  as  adversaries.  For  example, 
the  Air  Force  and  the  contractor  should  work  together  closely  to 
ensure  that  proposed  changes  in  regulations,  T.O.s,  etc.  are 
coordinated  as  early  as  possible.  (In  the  C-130  ATS,  this  was 
accomplished  by  MACOS  OL  Q  prior  to  PMRT  and  by  the  dedicated  on¬ 
site  group  in  the  34  TATG  thereafter.) 

Instructional  Systems  Development :  Individual  tryouts  should  be  at  the 
instructional  unit  (block)  level.  (In  the  C-130  ATS,  the  "unit"  is 
defined  as  a  collection  of  lessons  on  a  given  topic;  e.g., 
hydraulics.)  When  tryouts  involve  the  whole  unit,  one  should  check 
not  only  the  individual  lessons  but  how  well  they  fit  together  as 
an  instructional  unit.  In  courses  with  large  numbers  of  units  and 
even  larger  numbers  of  lessons,  this  approach  can  lead  to 
considerable  economies  in  the  use  of  available  student  test 
subjects  and  in  the  number  of  reviews.  In  addition,  in  existing 
courses  that  are  being  redesigned,  it  is  often  possible  to 
substitute  the  new  unit  for  an  old  (with  the  same  objectives)  ,  with 
minimal  disruption  of  student  flow. 

Student  Evaluation/Guarantee:  Student  guarantee  requires  that  all 
objectives  be  trained  and  tested  prior  to  the  flight  check.  The 
flight  check  covers  only  a  sample  of  the  training  objectives;  it 
should,  therefore,  be  viewed  as  one  part  of  the  "guarantee."  The 
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Air  Force  should  exercise  oversight  of  this  function  and  should 
periodically  spot-check  tc  ensure  that  all  objectives  are  being 
trained  and  tested  in  the  contracted  part  of  the  system. 

Test  and  Evaluation  Staffing:  It  is  important  to  use  trained  and 
experienced  personnel  for  test  and  evaluation.  Air  Force  flight 
examiner/stan-'Tval  experience  does  not  qualify  personnel  to  perform 
the  system  test  and  evaluation  functions. 

Marketing/ User  Participation:  The  Air  Force  should  ensure  that  the 
field  users  of  the  ATS  know  and  understand  that  the  system  is 
coming  and  why.  In  one  case,  for  example,  field  personnel  said 
that  they  were  not  going  to  use  the  new  5-point  flight  check  rating 
scale  because  it  was  not  specified  in  the  regulations. 

Personnel  Management:  Air  Force  military  personnel  responsible 
for  the  management  and  support  of  the  ATS  program  should  be  on 
controlled  tours  for  the  duration  of  the  development  program,  to 
minimize  program  turbulence  and  disruption. 

Test  and  Evaluation:  Test  and  evaluation  is  a  continuous  process 
which  evolves  throughout  the  life  of  the  program.  In  order  to 
maximize  its  benefits  for  both  the  Air  Force  and  contractor,  it 
should  be  planned,  manned,  and  initiated  early. 

Risk  Management/ Planning:  In  the  development  of  new  and/or 
particularly  innovative  programs,  one  should  manage  risk  by  having 
contingency  plans  and  fail-back  positions  available  to  preclude 
catastrophic  failures  during  the  development,  test,  and  operation 
of  the  training  system. 


C— 5  ATS 

C-5  ATS  Description 

The  C-5  ATS  is  a  system  of  personnel,  hardware,  software,  and 
courseware  that  produces  qualified  C-5  aircrew  members  and 
maintenance  engine  run  personnel.  The  system  currently  consists  of 
24  distinct  courses  for  the  MAC  formal  school,  in-unit  upgrade, 
maintenance  engine  run  qualification,  and  annual  simulator  profi¬ 
ciency  training.  It  includes  six  weapon  system  trainers  (WSTs) , 
four  cockpit  procedures  trainers  (CPTs) ,  two  cargo  door  and  cargo 
loading  part-task  trainers  (PTTs) ,  two  special  function  trainers 
(SFTs) ,  a  management  information  system  (MIS) ,  a  computer-aided 
instruction  (CAI)  system,  a  software  support  center,  and  a 
courseware  support  center.  This  integrated  system  will  ground- 
train  the  full  range  of  tasks  for  C-5  pilots  (except  aerial  refuel¬ 
ing  hands-on  part-task  training) ,  flight  engineers,  loadmasters, 
and  maintenance  engine  run  technicians.  The  end  products  are 
guaranteed  qualified  personnel.  The  contractor  is  responsible  for 
the  operation,  maintenance,  logistics  support,  and  configuration 
management  of  the  ATS. 
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C-5  ATS  Issues/Lessons  Learned 
Air  Force  Issues/Lessons  Learned 


Simulator! Procurement:  Consideration  should  be  given  to  the  use  of 
a  "best  commercial  practice"  rather  than  the  conventional  military 
specification  approach  for  simulator  procurements.  Using  best 
commercial  practice,  the  contractor  was  able  to  begin  installation 
of  the  first  of  six  C-5B  WSTs  only  20  1/2  months  after  contract 
award.  Twenty-three  months  after  contract  award,  the  WST  had 
undergone  Phase  II  Federal  Aviation  Administration  (FAA) 
certification  and  was  ready  for  training.  All  six  were  delivered 
within  29  months,  with  major  acquisition  cost  and  time  savings  to 
the  Government. 

User  Requirements:  All  users  should  be  considered  in  identifying 
training  requirements.  The  requirements  of  the  primary  user  are 
usually  obvious,  but  the  unique  requirements  of  other  agencies 
(e.g.,  the  Air  Force  Reserve,  the  Air  National  Guard,  other 
services,  and/or  foreign  governments)  who  send  trainees  through  the 
program  should  also  be  considered  and  incorporated  into  the  program 
if  feasible  and  cost-effective. 

Student  Guarantee! Evaluation:  A  4-hour  evaluation  at  the  end  of 
training  samples  only  a  small  percentage  of  the  total  number  of 
training  objectives.  Measures  obtained  during  the  course  of 
training  should  also  be  used  in  order  to  ensure  that  objectives  are 
met.  There  should  be  a  built-in  evaluation  process  which  provides 
timely  feedback  and  self-correcting  features  throughout  the  life 
cycle  of  the  system. 

Courseware  Readiness /Implementation:  "Turn-key"  startup  of  student 
training  under  a  contractor-developed  training  system  allows  the 
contractor  to  make  a  clean  break  from  the  old  methods  of  training 
to  new  contractor-provided  methods  and  media.  On  a  specified  day, 
the  previous  training  equipment  and  curricula  are  shut  down  and  the 
contractor-provided  training  system  is  turned  on.  However,  there 
can  be  disadvantages  to  that  approach  in  that  it  tends  to  ignore 
the  lessons  learned  in  years  of  operation  of  the  previous  training 
system.  Takeover  of  existing  courseware  and  training,  with  a 
requirement  to  develop  state-of-the-art  instruction,  may  be  an 
alternative.  The  contractor  can  develop  new  courseware  and  procure 
new  training  equipment  while  operating  the  existing  training 
program.  Lessons  learned  about  student  profiles,  academic  weak¬ 
ness,  subject  difficulty,  job  tasks,  etc.  while  conducting  the 
existing  training  could  be  incorporated  into  the  training  system 
during  development. 

Courseware  Updates:  One  goal  of  any  training  system  should  be  to 
provide  training  courseware  concurrently  with  procedural, 
equipment,  and  task  changes.  For  example,  if  major  procedural 
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changes  are  to  occur  next  Monday,  the  training  system  should  begin 
training  the  new  procedures  next  Monday,  not  6  months  from  then. 
Courseware  needs  ample  leadtime;  therefore,  training  system 
managers  need  to  work  out  with  all  affected  agencies  any  advance- 
notice  agreements  that  have  the  potential  to  impact  training. 
Likewise,  training  managers  need  contractual  tools  to  proceed 
quickly  with  changes.  Major  courseware  changes  require  dollars; 
dollars,  in  turn,  require  approval.  The  best  of  courseware 
maintenance  systems  cannot  keep  training  current  if  the  user  cannot 
cut  through  the  red  tape  to  approve  the  funding  changes.  In 
general,  user  organizations  should  commit  to  funding  and  approving 
training  changes  when  they  commit  to  making  a  procedural,  equip¬ 
ment,  task,  or  any  other  type  of  change  that  affects  the  training 
system.  At  that  point,  only  the  price  should  remain  to  be 
negotiated. 

Courseware  Configuration  Control:  Configuration  control  of  courseware 
documentation  is  absolutely  essential  in  any  large  training  system. 
In  the  C-5  ATS,  with  nearly  5,000  crew  performance  objectives 
(CPOs)  ,  it  is  essential  to  cross-reference  every  CPO  to  the 
courseware.  Likewise,  crew  tasks  must  be  cross-referenced  to  the 
lessons  where  they  are  taught,  and  a  lesson  index  must  list  all 
tasks  taught  within  all  lessons.  With  a  (single)  change  in  a  basic 
procedure,  it  is  possible  that  dozens  of  lessons  in  computer- 
delivered  training,  flight  simulators,  and  aircraft  flights  may 
have  to  be  revised.  Without  a  comprehensive  cross-reference  data 
retrieval  system,  the  affected  lessons,  student  handouts,  and 
visual  aids  cannot  easily  be  identified.  Without  the  cross- 
reference  system  in  the  C-5  ATS,  contractor  and  customer  quality 
control  would  be  a  nightmare. 

User  Acceptance:  Air  Force  management  should  strive  to  establish 
user  acceptance  (down  to  the  lowest  level)  of  a  training  system  in 
advance  of  actual  implementation  in  order  to  help  achieve  a 
critical  advantage.  This  can  be  done  through  newsletters,  road 
shows,  official  correspondence,  etc.  Perhaps  one  of  the  most 
effective  methods  to  spread  the  word  is  to  publish  a  user's  guide 
to  the  training  system,  complete  with  background  operational 
requirements  and  a  description  of  user  participation  in  modifi¬ 
cations  to  the  system.  Above  all,  one  should  tell  it  like  it  is 
and  listen  to  the  feedback. 

Facilities  Commonality:  If  more  than  one  site  is  being  developed, 
the  facilities  should  be  designed  with  as  much  standardization  as 
possible.  Thus,  if  a  delay  occurs  in  facility  construction  at  one 
site,  equipment  can  be  diverted  and  installed  at  another  without 
extensive  site  modification. 

Training  Task  Analysis:  Air  Force  personnel  involved  in  training 
system  development  should  approach  a  contractor-conducted  training 
task  analysis  with  an  open  mind.  They  should  understand  that  there 
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are  some  tasks  which  have  been  trained  historically — not  because 
there  was  a  string  rationale  for  doing  so  but  merely  because  it  had 
always  been  done  that  way.  A  thorough  task  listing/ evaluation 
standards  document  can  provide  a  much  better  definition  of  actual 
training  requirements.  In  contrast,  they  should  also  be  aware  that 
some  tasks  trained  in  the  old  system  were  there  because  of  lessons 
learned  the  hard  way.  This  wisdom  should  be  incorporated  into  the 
new  system  also.  Thus,  the  Air  Force  user  should  stand  up  and  be 
heard  to  ensure  that  his  experience  is  utilized. 

Computer-Based  Training  (CBT)  System:  In  the  interest  of  not  overly 
constraining  the  contractor,  the  C-5  ATS  system  did  not  state  a 
limiting  response  time  to  advance  from  one  CBT  screen  of  text/ 
graphics  to  the  next.  Consequently,  after  several  months  of 
operation,  the  contractor  is  still  trying  to  fix  the  system  so  that 
it  achieves  a  response  time  that  is  acceptable  to  students. 
Certain  constraints  need  to  be  specified. 

Contractor  Issues/Lessons  Learned 

CBT:  Bidders  should  not  be  required  to  lock-in  a  particular 
CBT  system  prior  to  contract  award.  That  approach  may  make  it 
easier  for  the  source  selection  team  to  compare  proposals,  but  it 
also  forces  the  bidder  to  select  a  specific  hardware/ software 
configuration  too  far  in  advance  of  determining  the  particular 
needs  of  the  training  system.  In  addition,  the  state  of  the  art  in 
CBT  is  changing  so  rapidly  that  some  future  opportunities  could  be 
missed.  For  example,  the  contractor  selected  a  particular  CBT 
system  for  the  C-5  and  now  feels  that  it  might  have  been  better  if 
they  could  have  waited  until  after  contract  award  to  make  their 
final  selection.  The  acquisition  agency  could  provide  some  general 
functional  requirements  and  some  representative  system  and  cost 
guidelines  for  competition  that  would  permit  the  contractor  to 
change  his  original  selection  after  contract  award  if  a  more  cost- 
effective  system  or  approach  were  identified. 

Task/Lesson  Commonality:  The  common  items  on  the  master  task 
listing  should  be  integrated  across  crew  positions.  This  list  of 
common  items  might  provide  the  opportunity  to  use  common  lessons  or 
lesson  modules  for  the  different  position  courses. 

Simulator/ Procurement:  The  Government  should  not  require  the 
military  specification  approach  for  simulator  procurement. 
Instead,  they  should  permit  "best  commercial  practice."  This 
approach  was  used  in  the  C-5  ATS,  and  it  not  only  greatly  cut  the 
development  time  but  also  produced  the  first  Phase  II  certified 
simulator  in  the  Air  Force  inventory. 

Student  Guarantee:  It  is  smart  to  require  a  guaranteed  student 
product  in  order  to  provide  an  anchor  point  for  product  definition 
and  acceptance  testing.  However,  to  ensure  that  everything  is 
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taught  to  every  student,  the  flight  examiner  should  be  given  the 
flexibility  to  construct  his  individual  checks  such  that  anything 
on  the  master  task  listing  may  be  included. 

Student  Performance  Requirements:  The  Air  Force  must  specify  what  the 
product  should  be  (i.e.,  performance  criteria).  Despite  the 
difficulty  associated  with  preparing  an  explicit  specification  for 
many  kinds  of  critical  tasks,  there  is  a  need  to  work  out  a  new 
approach  and/or  a  common  understanding  between  the  Air  Force  and 
the  contractor  concerning  how  to  handle  certain  hard-to-measure 
areas. 

Procurement  Leadtime:  The  timely  procurement  of  long-leadtime 
items  from  the  aircraft  vendor  for  use  in  the  simulator  is  critical 
for  concurrent  simulator  development  in  new  systems  (e.g.,  seats, 
control  columns,  rudder  pedals) .  There  is  a  need  for  a  minimum  of 
two  ship-sets.  This  could  be  a  real  problem  for  the  C-17  and  is  an 
ongoing  problem  for  all  ATSs.  There  is  a  need  to  order  parts  as 
soon  as  an  aircraft  change  is  announced.  Each  simulator  should  be 
tied  to  an  aircraft  tail  number,  so  that  all  changes  that  affect 
the  aircraft  result  in  an  automatic  updating  of  the  corresponding 
simulator.  This  is  the  approach  used  by  United  Airlines  in  its 
training  facility. 

Air  Force  Subject  Matter  Experts  (SMEs):  The  availability  and  continuity 
of  aircrew  SMEs  is  very  important  during  the  development  of  a  new 
ATS.  However,  it  was  noted  that  the  Air  Force  was  concerned  that 
United  Airlines  might  take  the  SMEs'  recommendations  as  official 
Air  Force  approval. 

On-Site  Development:  Insisting  on  an  on-site  development  team  may 
affect  the  quality  of  the  program  because  the  contractor's  best 
people  may  not  want  to  go  to  an  unattractive  and/or  expensive  area. 
(Authors'  note:  In  the  opinion  of  the  authors,  on-site  development 
is  usually  the  preferred  approach.  However,  in  practice  there  have 
been  many  cases,  including  the  C-141  and  C-5,  where  the  initial 
development  of  instructional  materials  was  accomplished  at  the 
contractor's  facilities.  In  these  instances,  the  Air  Force  has 
assigned  a  small  cadre  of  personnel  to  work  in-plant  with  the 
contractor  and  has  sent  other  SMEs  to  support  specific  reviews  on 
an  as-needed  basis.) 

Procurement  Process:  The  procurement  process  should  be  made  more 
responsive  so  that  it  can  react  more  rapidly  to  major  changes  in 
the  ATS  that  result  from  changes  by  the  Air  Force.  For  example, 
during  procurement  of  the  C-5  ATS,  C-5s  were  introduced  into  Air 
National  Guard  units  at  Westover  AFB,  Maine;  Stewart  AFB,  New  York; 
and  Kelly  AFB,  Texas.  It  was  not  possible  to  modify  the  basic 
program  in  time  to  include  these  new  requirements  as  a  part  of  the 
C-5  ATS  development.  As  a  consequence,  Guard  personnel  must  go  to 
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Altus  AFB,  Oklahoma,  for  initial  qualification  in  the  C-5.  This 
imposes  a  hardship  on  Guard  personnel  with  other  full-time  jobs. 

Personnel  Continuity :  One  of  the  major  (potential)  benefits  of  a 
contractor-supported  training  system  is  the  continuity  of  the 
contractor  staff,  for  it  counteracts  the  high  personnel  turnover 
which  occurs  in  the  military. 

KC-10  ATS 


KC-10  ATS  Description 

The  KC-10  ATS  is  designed  to  provide  training  for  KC-10 
pilots,  flight  engineers,  and  boom  operators.  The  contractor 
provided  a  total  ground  training  package  consisting  of  simulator 
acquisition  and  maintenance,  training  system  design  and  management, 
and  delivery  of  academic  and  simulator  instruction.  Training  is 
provided  at  Barksdale  AFB,  Louisiana;  March  AFB,  California;  and 
Seymour  Johnson  AFB,  North  Carolina.  The  contractor-supported 
training  system  was  delivered  with  the  aircraft,  and  there  never 
has  been  an  Air  Force  counterpart  for  that  portion  of  training 
conducted  by  the  contractor.  The  contractor  provides  ground 
training  for  initial  qualification  and  continuation  training;  the 
Air  Force  provides  in-flight  instruction,  administers  check  rides, 
and  provides  mission  qualification  training.  Initial  qualification 
training  occurs  in  phases.  In  Phase  I,  the  contractor  provides  all 
academic  instruction,  followed  by  all  simulator  training.  The 
contractor  certifies  that  the  student  has  met  all  training 
objectives  to  that  point  and  that  the  student  will  pass  an  in¬ 
flight.  SAC  Regulation  60-4  (Standardization/Evaluation  Program) 
check  ride.  (Authors'  note:  Air  Force  instructors  administer  a 
60-4  check  in  the  simulator  prior  to  flight  training.)  Delivery  of 
instruction  involves  limited  CBT,  but  training  management  is 
accomplished  almost  exclusively  with  computers.  In  Phase  II,  Air 
Force  instructors  provide  in-flight  instruction  and  administer  the 
60-4  check  ride.  If  at  any  time  during  this  sequence  the  student 
fails  to  meet  requirements,  the  contractor  provides  additional 
training  at  no  additional  cost  to  the  Government.  There  is  a 
penalty  clause  for  late  graduation,  but  it  has  never  been 
exercised.  (Authors'  note:  The  Air  Force  and  the  contractor  have 
established  a  system  to  jointly  review  and  resolve  potential 
student  problems  prior  to  the  flight  phase.  As  a  result,  there 
have  been  no  flight  check  failures  or  late  graduations  which  were 
attributable  to  the  contractor's  failure  to  perform.) 

KC-10  ATS  Issues/Lessons  Learned 

Air  Force  Issues/Lessons  Learned 

The  Air  Force  training  personnel  contacted  at  Barksdale  AFB, 
Louisiana,  appeared  to  be  quite  pleased  with  the  contractor's 
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performance  and  voiced  no  complaints  about  the  current  program.  In 
addition,  they  appeared  to  be  very  satisfied  with  what  they 
characterized  as  an  atmosphere  of  mutual  trust  and  cooperation 
between  the  Air  Force  and  the  contractor. 

Contractor  Issues/Lessons  Learned 

Student  Performance  Requirements:  The  Air  Force  should  (a)  define 
student  entry  skills,  (b)  define  what  they  want  students  to  do  when 
trained,  and  (c)  specify  how  much  time  is  available  for  training.  .  . 
"but  should  not  tell  the  contractor  how  to  do  it!"  Then  if  the  Air 
Force  has  a  problem  with  the  contractor's  approach,  the  two  should 
discuss  it. 

Simulator/ Procurement:  Simulators  should  be  acquired  using  "best 
commercial"  practice  rather  than  the  military  specification 
approach.  (Authors'  note:  The  additional  effort,  cost,  and  time 
necessary  to  meet  the  administrative,  data,  and  review  requirements 
imposed  by  current  military  specifications  and  standards  do  not 
produce  any  real  value-added  to  the  resulting  training  simulators. 
Both  the  flight  simulator  industry  and  the  underlying  technology 
are  relatively  mature  and  reliable;  so,  there  is  little  or  no  risk 
in  a  "best  commercial  practice"  approach.) 

Aircraft  Data:  It  is  often  difficult  for  an  ATS  contractor  to 
obtain  information  from  the  aircraft  manufacturer;  at  such  times, 
Air  Force  influence  must  be  used  to  expedite  the  process. 
(Authors'  note:  The  difficulty  in  obtaining  data  is  a  function  of 
the  ATS  contractor  involved.  In  those  instances  where  the  ATS 
contractor  is  also  a  major  customer  of  the  aircraft  vendor,  the 
contractor  probably  has  at  least  as  much  influence  as  the  Air  Force 
itself. ) 

Source  Selection:  One  opinion  voiced  strongly  was  "Go  for  the  best 
contractor  at  source  selection  and  not  lowest  bidder!" 

Source  Selection:  There  should  be  one  contractor  who  is 
responsible  for  everything  (i.e.,  training  development,  simulator 
operation,  maintenance,  etc.). 

Some  General  Issues/Lessons  Learned 

The  following  lessons  learned  were  contributed  by  one  of  the 
authors.  Dr.  Robert  Nullmeyer.  They  are  based  on  his  collective 
experiences  as  program  manager  for  the  C-130  MATS  study,  as  well  as 
his  involvement  with  a  number  of  other  proposed  and  ongoing 
programs  including  the  C-5  ATS,  C-130  ATS,  B-52/KC-135  CCTS 
Modernization,  and  the  SOF  ATS. 
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Source  Selection:  One  should  not  select  a  contractor  simply  on  the 
basis  of  the  lowest  bid.  One  must  thoroughly  review  the  technical 
proposal  and  personnel  qualifications  to  ascertain  that  the  bidder 
has  the  understanding  and  capability  to  do  the  job. 

Instructions  to  Offerers  (ITOs):  ITOs  should  be  structured  so  that  the 
level  of  understanding  of  the  bidder (s)  can  be  determined  in  the 
proposal.  There  should  be  enough  information  included  in  the  ITOs 
to  explain  precisely  what  the  user  wants  to  see  in  the  proposal, 
keeping  in  mind  that  the  proposal  is  the  primary  data  source  for 
determining  a  prospective  contractor's  understanding  of  the 
requirements  and  capability  to  deliver  the  desired  product.  The 
RFP  for  the  C-130  ATS  tied  ITOs  to  specific  paragraphs  in  the  SOW 
and  requested  a  discussion  of  how  specific  SOW  requirements  would 
be  met.  It  is  an  excellent  model! 

System/ Subsystem  Integration:  It  is  imperative  that  all  development 
programs  provide  adequate  management,  systems  engineering,  and 
technical  direction  to  ensure  the  optimal  integration  of  components 
into  subsystems,  and  subsystems  into  a  total  system.  This  is 
particularly  critical  for  activities  that  cut  across  a  number  of 
system  components  and/or  functions  (e.g.,  the  training  management 
system  and  the  system  test  and  evaluation  program) .  There  is  a 
tendency  in  some  programs  for  interrelated  components  and 
subsystems  to  be  designed  and  developed  in  relative  isolation  from 
each  other,  particularly  when  driven  by  the  pressures  of  production 
schedules.  Therefore,  there  is  a  critical  need  to  establish  a 
formal  organizational  function  with  the  responsibility,  authority 
and  resources  to  exercise  technical  and  managerial  oversight  in 
ord^r  to  ensure  proper  integration  at  all  levels. 

Training  Design  Concept:  Air  Force  training  systems  must  be 
designed  to  train  for  both  peacetime  and  wartime  operations,  as 
inadequate  performance  in  either  case  may  be  fatal. 

Student  Guarantee:  Within  a  proficiency-based  system,  data  should 
be  collected  throughout  training  to  ensure  that  objectives  are  met. 
A  single  check  ride  covers  a  limited  sample  of  behaviors  and,  as 
such,  is  but  one  piece  of  evidence  to  be  considered  with  other 
available  data  in  assessing  student  capabilities. 

System/ Program  Evaluation:  Experience  to  date  suggests  that  many 
contractors  appear  to  have  an  adequate  understanding  of  the 
traditional  student/crew  evaluation  concepts  but  not  of  program 
evaluation. 

Risk  Management:  There  obviously  should  oe  a  specific  plan  and 
schedule  for  program  development  and  implementation.  However,  the 
contractor  should  also  have  a  contingency  plan  that  identifies  what 
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optional  approaches  will  be  available  to  minimize  risk  in  the  event 
that  technical,  schedule,  etc.,  problems  do  occur. 

Indefinite  Quantities  Contracts  ( IQCs ):  Use  of  an  existing  IQC  for  FEA 
can  save  time  and,  perhaps,  dollars  if  the  available  contractor  has 
the  capability  to  do  the  job.  In  any  event,  the  FEA  requirements 
should  be  tailored  to  what  the  IQC  contractor  can  reasonably 
accomplish  with  the  available  resources. 

TTCP  Survey 

At  the  time  this  section  was  prepared,  only  four  responses  to 
the  TTCP  survey  had  been  received.  The  responses  included  were 
from  the  UTP-2  representatives  from  the  Australian  Army  and 
Canadian  National  Defence  Headquarters  (two  inputs) .  These  inputs 
are  summarized  below.  (Authors'  note:  An  input  was  also  received 
from  the  U.S.  Naval  Training  Systems  Center  (NTSC)  representative; 
it  included  a  description  of  the  E-6A  program.  However,  because  of 
recent  changes  in  this  program,  the  authors  elected  to  defer 
discussing  it  until  additional  information  can  be  obtained  and 
verified.  Because  the  E-6A  is  a  unique  effort  and  a  potentially 
rich  source  of  valuable  lessons  learned,  it  will  be  treated  in 
greater  detail  in  a  later  report.) 

Australian  Army 

The  Australian  Army  representative  stated  that  at  the  present 
time  there  is  only  one  instance  of  contractor-supported  aircrew 
instruction  in  the  Australian  Army.  They  are  employing  1  civilian 
flight  instructor  from  each  of  2  airlines  to  train  6  pilots  to 
achieve  proficiency  on  fixed-wing  Nomad  light  observation  aircraft. 
Both  instructors  are  assigned  to  the  Army  School  of  Aviation  and 
report  to  the  Senior  Instructor  of  that  wing  in  the  same  manner  as 
the  Army  flying  instructors  on  staff.  These  instructors  were 
employed  to  release  Army  staff  to  train  increasing  numbers  of 
pilots  and  to  "establish  continuity  and  expertise  in  the  training 
of  foreign  personnel.”  he  noted  that,  "To  date  the  use  of  the 
contracted  civilian  instructors  has  been  successful.  This  success 
has  been  achieved  because  ex-Army  instructors  have  been  employed 
and  the  contracted  instructors  are  integrated  as  members  of  the 
instructional  staff." 

The  Australian  Army  has  recently  initiated  a  cost-benefit 
study  to  assess  the  potential  advantages  of  increasing  the  use  of 
contractors  to  train  Army  pilots.  The  requirement  for  the  study 
was  prompted  by  a  substantial  increase  in  the  number  of  the  Army's 
inventory  of  rotary  wing  aircraft  resulting  from  a  large  buy  of 
Blackhawk  helicopters.  The  U.S.  Army  is  currently  using  civilian 
contract  instructors  for  basic  helicopter  flying  training  at  Ft. 
Rucker,  Alabama. 
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Canadian  Forces 


The  Canadian  Forces  (CF)  have  an  out-service  training  (OST) 
policy  which  states,  "All  individual  training  shall  be  conducted 
within  the  DND  (Department  of  National  Defence)  unless  it  is  more 
appropriate  and  economical  to  obtain  training  outside  the 
department."  However,  as  is  the  case  with  the  USAF,  the  CF  are 
looking  more  and  more  to  contract-supported  training  to  reduce  the 
demands  on  its  military  personnel  resources  to  support  training 
programs.  In  fact,  the  Primary  Flying  Training  (PFT)  Study  Team 
was  formed  in  early  1989  to  investigate  the  contracting-out  of  CF 
flying  training. 

The  CF,  like  the  USAF,  has  a  long  history  of  sending  personnel 
to  factory  training  courses  (i.e.,  Type  1,  Contract  Special 
Training)  for  initial  maintenance  and  operator  training, 
particularly  for  new  systems.  However,  to  date  the  CF  have 
virtually  no  experience  with  respect  to  a  civilian  contractor- 
supported  aircrew  training  system  except  for  the  C-144  Challenger 
program,  which  is  described  in  the  following  paragraphs. 

C-144  Aircrew  Training  Program.  C-144  Program  Description:  The  C-144 
Challenger  is  a  12-passenger  executive  jet.  Because  the  aircraft 
was  purchased  from  a  Canadian  source  (with  ready  access  to  the  unit 
employing  the  aircraft),  second-  and  third-line  (i.e.,  shop  and 
depot )  maintenance  have  been  contracted  to  the  original  equipment 
manufacturer  (OEM) .  Military  personnel  have  been  restricted 
to  first-line  servicing  and  black  box  replacement.  Because 
the  Challenger's  civilian  and  military  roles  are  identical,  CF 
pilots  are  being  trained  in  a  commercial  pilot  conversion 
course  provided  by  Flight  Safety  International.  The  course 
is  2 -weeks  in  length  and  includes  both  ground  school  instruction 
and  flight  simulator  training.  The  course  is  complemented  by 
an  additional  in-service  ground  school  and  flying  training 
program  which  focuses  on  military  procedures  and  standards.  To 
maintain  certification  in  the  aircraft,  CF  pilots  return  to 
the  contractor  each  year  for  flight  simulator  refresher  training 
and  aircraft  operating  instruction  review.  According  to  the 
UTP-1  representative,  the  benefits  realized  from  this  program 
include : 

a.  A  reduction  in  the  person-year  (PY)  requirement  to  support 
the  fleet  (for  both  the  training  and  maintenance  programs)  of  120 
PYs ; 


b.  CF  support  to  Canadian  industry  which  has  enabled  the 
industry  to  expand  into  a  major  world-class  aircraft  manufacturer 
and  support  organization. 

C-144  Issues  /Lessons  Learned:  The  following  "lessons  learned"  from 
the  Challenger  OST  program  were  identified: 
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a.  Contractors  have  been  willing  to  amend  commercial  course 
content  to  meet  unique  CF  requirements. 

b.  Commercial  training  agencies,  which  have  Canadian  Ministry 
of  Transport  or  FAA  certification,  provide  training  at  a  high 
standard  of  quality. 

c.  Availability  of  courses  can  be  contractually  guaranteed. 

d.  Course  schedules  can  be  made  to  meet  CF  requirements  (if 
they  are  decided  upon  in  advance  and  included  in  the  contract) . 

e.  Sole-source  contractors  have  been  responsible,  in  that  they 
have  not  charged  unreasonable  rates. 

f.  If  personnel,  for  whatever  reason,  are  unavailable  at  the 
last  minute  to  undergo  training,  or  if  the  department  wishes  to 
amend  course  schedule  or  content,  the  department  must  be  prepared 
to  meet  contractual  obligations/penalties  accordingly. 

g.  Ready  access  to  commercial  industry's  lessons  learned  and 
experiences  has  improved  the  CF's  knowledge  base. 

h.  A  close  working  relationship  with  the  aircraft  manufacturer 
has  kept  the  CF  abreast  of  the  latest  modifications  and 
advancements  to  the  aircraft. 

i.  The  reluctance  to  accept  the  fact  that  commercial  industry 
can  train  military  personnel  to  an  equally  high  standard  is 
gradually  being  eroded. 

j .  Limiting  maintenance  personnel  to  solely  a  first-line 
servicing  role  has  incurred  some  morale  problems  in  that  their 
ability  to  advance  their  experience  and  knowledge  is 
correspondingly  limited. 


DISCUSSION 
General  Discussion 


Perhaps  the  most  important  lesson  learned  from  a  review  of 
major  ATS  programs  to  date  is  that  contracting-out  does  not  relieve 
the  military  of  responsibility  for  that  part  of  the  program  under 
the  contractor's  control.  One  original  assumption  which  underlaid 
the  promulgation  of  the  Air  Force  concept  of  Total  Contractor 
Training  was  that  the  Air  Force  (or  any  service)  could  simply 
define  its  requirements  in  terms  of  aircrew  capability,  student 
throughput,  etc.,  and  then  stand  back  and  allow  the  contractor  to 
set  up  a  training  factory  to  produce  a  "guaranteed  student"  who 
could  be  accepted  or  rejected  by  the  Air  Force  on  the  basis  of  a 
single  flight  check.  This  hardware  acquisition  metaphor  reflected 
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some  very  serious  misconceptions  with  respect  to  the  nature  and 
purpose  of  military  aircrew  training,  the  limits  on  the  omniscience 
of  contractors,  and  the  ability  of  the  Air  Force  (or  anyone  else) 
to  write  an  unambiguous  aircrew  performance  specification  which 
could  be  satisfied  by  a  third  party  without  any  Air  Force 
involvement  or  oversight.  Fortunately,  most  of  these 
misconceptions  are  being  dispelled  as  a  result  of  the  experience 
being  gained  with  existing  systems,  although  not  without  cost. 

There  is  a  growing  awareness  that  the  successful  development 
and  operation  of  an  ATS  is  a  team  effort.  It  is  beccming 
recognized  that  it  is  in  the  best  interest  of  the  Air  Force  to  work 
with  the  contractor  throughout  the  ATS  life  cyrlo  to  ensure  that 
the  system  does  not  fail.  The  contractor  can  walk  away  from 
failure,  but  the  Air  Force  cannot!  There  is  also  a  growing 
awareness  that  contractors,  unlike  diamonds,  are  not  "forever." 
Several  of  the  existing  ATSs  have  had  to  be  recompeted  before  the 
original  programs  had  run  their  full  course.  This  clearly 
indicates  that  the  Air  Force  cannot  simply  stand  back  and  allow  the 
contractor  to  do  his  thing  without  regard  for  how  he  does  it  in  the 
contracted  part  of  the  system.  For,  no  matter  who  develops  or 
operates  the  ATS,  the  military  customer  must  know  enough  about  the 
system  baseline  to  be  able  to  transition  to  a  new  contractor  if  it 
becomes  necessary,  at  an  acceptable  cost  and  with  minimum 
disruption  to  the  total  training  program. 

Future  Plans 


The  near-term  plans  for  follow-on  activity  under  this  effort 
include  expanding  the  core  systems  database  by  the  addition  of 
information  on  the  C-141  and  SOF  ATSs.  During  FY91,  a  detailed 
review  and  analysis  of  the  training  management  and  evaluation 
components  of  the  core  systems  will  be  conducted.  The  information 
obtained  from  this  effort  will  be  used  to  support  the  development 
of  a  set  of  guidelines  for  the  identification  of  training  system 
information  and  evaluation  guidelines  for  use  by  acquisition 
agencies  and  using  commands. 

The  lessons  learned  identified  in  this  document,  as  well  as 
those  to  be  collected  during  the  next  phase  of  this  effort,  will 
also  be  provided  in  report  form  to  the  Air  Force  Logistics  Command 
for  inclusion  in  the  training  section  of  its  existing  lessons 
learned  database.  In  addition,  the  same  information  will  be 
submitted  to  the  DoD  Training  and  Performance  Data  Center  (TPDC) 
for  consideration  as  a  possible  nucleus  entry  for  a  new  ATS  lessons 
learned  subset  of  its  current  database. 
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To:  Members  of  UTP-2 
From:  Wayne  Waag 

Subject:  Issues  in  Aircrew  Contracted  Instruction 

1.  At  the  last  UPT-2  meeting  we  discussed  a  point  paper  prepared  by 
Dr.  Robert  Nullmeyer  of  the  Operations  Training  Division  of  the  Air 
Force  Human  Resources  Laboratory.  This  paper  identified  a  number 
of  issues  resulting  from  the  introduction  of  contractor-  supported 
aircrew  training  in  the  U.S.  Air  Force  (See  Attachment  1) .  Prior 
to  the  conclusion  of  the  meeting,  it  was  agreed  that  contract 
instruction  was  an  area  of  mutual  concern  and  that  a  collaborative 
effort  should  be  pursued.  It  was  also  agreed  that  I  would  accept 
responsibility  for  revising  the  original  list  of  issues  and 
soliciting  additional  inputs  from  other  Panel  Members  for 
discussion  at  our  next  meeting. 

2.  Attachment  2  contains  a  listing  of  some  of  the  major  issues 
identified  by  personnel  involved  in  ongoing  or  proposed  contractor- 
supported  aircrew  training  programs  in  the  U.S.  Air  Force.  This 
list  was  compiled  with  the  support  of  Dr.  Marty  Rockway  of  the 
University  of  Dayton  Research  Institute  on-site  staff  at  AFHRL/OT. 
Dr.  Rockway  is  the  principal  investigator  on  a  task  to  develop  an 
"Aircrew  Training  System  Design  and  'Lessons  Learned'  Database."  A 
summary  of  this  task  is  included  as  Attachment  3 . 

3.  It  is  requested  that  you  provide  me  with  the  following 
information  for  consolidation  in  a  draft  paper  which  will  be  used 
as  the  focus  for  further  discussion  at  the  next  Panel  meeting: 

a.  A  brief  description  of  the  major  contractor-supported 
training  programs/ systems  implemented,  under  development,  and/or 
proposed  in  your  particular  military  department  or  branch  of 
service.  Please  identify  the  kinds  of  functions/services  provided 
by  the  contractor  (s)  .  It  would  also  be  helpful  if  the  rationale  for 
contracting  out  instruction/training  could  be  identified.  Use  the 
example  provided  in  Attachment  4  for  guidance  concerning  the  kind 
of  information  and  the  level  of  detail  required  at  this  time. 

b.  A  listing  and  brief  description  of  the  major  training  design, 
development,  implementation,  evaluation  and/or  management  issues 
associated  with  the  actual  or  proposed  introduction  of  contractor 
supported  training. 

c.  A  brief  description  of  any  "lessons  learned"  which  might  be 
applied  to  other  ongoing  or  proposed  contractor  supported 
instruction/training  programs. 

4.  Please  send  your  inputs  to.... 
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POINT  PAPER 


ISSUES  IN  AIRCREW  CONTRACTED  INSTRUCTION 


Robert  T.  Nullmeyer 
Operations  Training  Division 
Air  Force  Human  Resources  Laboratory 
Williams  AFB,  Arizona  85240-6457 


There  is  a  continuing  trend  within  the  USAF  toward  the  use  of 
contract  resources  in  support  of  aircrew  training.  Contract 
resources  have  been  used  in  a  variety  of  applications  ranging  from 
specific  courseware  design  to  total  turnkey  training  system  design 
and  operation.  Examples  include  the  KC-10  training  program,  the  C- 
5  Aircrew  Training  System,  and  the  recently  awarded  C-130  Aircrew 
Training  System.  To  date,  most  applications  have  occurred  for 
transport  aircraft  in  which  the  training  bears  much  in  common  with 
the  airline  industry.  For  tactical  applications,  contract 
resources  have  been  used  only  in  the  design  of  training  curricula 
and,  in  limited  instances,  for  the  conduct  of  simulation  training. 
Given  this  trend  toward  greater  use  of  contractor  resources,  there 
appears  a  need  to  briefly  examine  some  of  the  underlying  issues 
involving  contract  instruction.  These  include: 

Contractor  vs.  Air  Force  Roles  and  Responsibilities.  The  major 
components  of  a  training  system  are:  overall  training  system 
design;  courseware  development  and  maintenance;  delivery  of 
instruction  (academics,  simulators,  flight) ;  maintenance  of 
training  devices,  including  spare  parts;  and  training  system 
management  support  including  scheduling  and  data  processing.  An 
early  decision  is  required  regarding  which  of  these  components  will 
be  contracted  and  which  will  remain  organic. 

Program  Goals.  Why  is  a  training  support  contract  being 
contemplated?  What  are  the  anticipated  benefits  of  contracted 
instruction?  How  will  program  costs  be  amortized?  What  is  the 
government  actually  purchasing? 

Training  System  Design. 

1.  Are  there  clearly  superior  concepts  and  strategies?  For 
example,  what  are  the  strengths  and  weaknesses  of  phase  of  flight 
vs.  systems-oriented  training?  If  there  are  demonstrable 
differences,  what  are  they,  and  what  are  the  boundary  conditions? 

2.  How  do  cognitive  aspects  (e.g.,  decision  making, 
leadership,  and  situational  awareness)  enter  into  the  design 
process  and  how  should  these  be  trained? 
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Attachment  1 


3.  What  are  the  limitations  of  component  technologies  like 
CBT  and  simulators?  Given  current  state-of-the-art  training 
devices,  is  there  a  requirement  for  flight? 

4.  What  training  activities  should  be  centralized  in  a  formal 
school  and  what  requirements  are  better  met  at  gaining  units? 

5.  Is  an  airline  model  appropriate  for  military  training? 

Guaranteed  Student  or  Crewmember  Performance  Level.  The  Air  Force 
is  far  from  unified  concerning  the  nature  of  contracted  training 
support.  On  the  high  end,  the  government  specifies  performance 
requirements  for  training  system  graduates  and  the  contractor  is 
given  freedom  and  responsibility  to  reach  this  objective.  KC-10, 
C-5 ,  and  C-130  formal  schools  fall  into  this  category  of  contracted 
support.  A  number  of  issues  specific  to  student  performance 
guarantees  are  emerging: 

1.  How  should  training  requirements  be  specified?  Both  the 
traditional  four-point  scales  and  the  applicable  Air  Force 
regulations  appear  inadequate. 

2.  What  is  the  proper  level  of  government  oversight  regarding 
contractual  compliance,  courseware  quality  assurance,  and  training 
device  control? 

Improved  Technologies  and  Concepts.  How  can  the  government 
encourage  the  capital  investments  required  to  improve  training 
systems  beyond  minimum  requirements? 

Training  System  Evaluation.  Why  should  training  systems  be 
evaluated?  What  objectives,  methodologies,  and  measures  are 
required  to  adequately  evaluate  training  systems  at  design, 
development,  implementation  and  continuing  operation  stages  of  the 
program?  Which  elements  need  to  be  included  in  training  system 
test  and  evaluation?  How  complete  should  coverage  be? 

If  there  is  insufficient  interest  among  the  member  nations,  it 
is  recommended  that  an  effort  be  initiated  to  address  these  issues 
in  depth  with  the  purpose  of  establishing  a  set  of  guidelines  for 
use  by  training  organizations  contemplating  the  use  of  contracted 
instruction.  This  would  be  accomplished  through  an  evaluation  of 
case  studies  of  member  nations'  experiences  with  contracted 
training.  Based  on  lessons  learned  from  previous  applications,  a 
set  of  guidelines  would  be  established  identifying  issues  to  oe 
considered  before  a  decision  is  made  regarding  use  of  contractor 
resources . 
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SOME  ADDITIONAL  ISSUES  IN  AIRCREW  CONTRACTED  INSTRUCTION/TRAINING 

The  following  paragraphs  identify  a  number  of  issues  generated 
by  the  introduction  of  contractor-supported  aircrew  training  in  the 
U.S.  Air  Force.  The  issues  listed  below  are  intended  to  augment 
those  discussed  in  Dr.  Nullmeyer's  point  paper  which  is  included  as 
Attachment  1.  These  particular  issues  are  a  subset  of  a  larger 
group  identified  during  the  course  of  gathering  information  for  the 
"Aircrew  Training  System  Design  and  Lessons  Learned  Database" 
described  in  Attachment  3.  Many  of  the  other  issues  identified 
under  that  project  to  date  are  felt  to  have  relatively  little 
generality  beyond  their  specific  Air  Force  context. 


1.  WHAT  IS  THE  RATIONALE  FOR  CONTRACTING  OUT  AIRCREW  INSTRUCTION/ 
TRAINING?  WHAT  GUIDELINES/PRINCIPLES/FACTORS  ARE  USED  TO  DETERMINE 
WHAT  TRAINING  FUNCTIONS /PHASES  WILL  BE  PERFORMED  BY  A  CONTRACTOR? 

2.  WHAT  ARE  THE  CONSIDERATIONS,  CONSTRAINTS,  AND  DIFFERENCES  IN  THE 
UTILIZATION  OF  CONTRACTOR  TRAINING  FOR  DIFFERENT  KINDS  OF  AIRCRAFT 
AND  MISSIONS  (E.G.,  BOMBERS,  TRANSPORTS,  FIGHTERS)? 

3.  WHAT  ARE  APPROPRIATE  LEVELS  OF  MILITARY  INVOLVEMENT/MANAGEMENT 
FOR  VARIOUS  KINDS  OF  CONTRACTOR- SUPPORTED  TRAINING?  WHAT  ARE  THE 
FACTORS  THAT  DETERMINE  THE  DEGREE  OF  MILITARY  INVOLVEMENT  AND/OR 
MANAGEMENT  OF  THE  CONTRACTOR-SUPPORTED  PORTIONS  OF  THE  TRAINING 
PROGRAM?  HOW  DOES  THE  MILITARY  AND  CONTRACTOR  MANAGEMENT  STRUCTURE 
CHANGE  AT  VARIOUS  STAGES  IN  THE  TRAINING  SYSTEM  LIFE  CYCLE? 

4.  WHAT  MECHANISMS  SHOULD  BE  SET  UP  FOR  RESOLVING  DIFFERENCES  OF 
OPINION  BETWEEN  THE  MILITARY  AND  CONTRACTOR?  (FOR  EXISTING 
SYSTEMS,  "WHAT  MECHANISMS  HAVE  BEEN  SET  UP  AND  HOW  SUCCESSFUL  HAVE 
THEY  BEEN?") 

5.  WHAT  (IF  ANY)  KIND  OF  STUDENT  PERFORMANCE  GUARANTEE  SHOULD  BE 
WRITTEN  INTO  THE  CONTRACT?  HOW  DOES  THE  MILITARY  INSURE  THAT  THE 
GUARANTEE  HAS  BEEN  MET?  SHOULD  A  PENALTY  (INCENTIVE)  CLAUSE  FOR 
NON-COMPLIANCE  BE  INCLUDED?  (HAS  SUCH  A  CLAUSE  EVER  BEEN  EXERCISED 
IN  AN  EXISTING  CONTRACTOR-SUPPORTED  SYSTEM?) 

6.  DOES  A  STUDENT  GUARANTEE  CONSTRAIN  THE  DESIGN  OF  THE  TRAINING 
SYSTEM?  FOR  EXAMPLE,  WHAT  PROBLEMS/INEFFICIENCIES  MAY  BE 
INTRODUCED  BY  PHASE  OF  FLIGHT,  FUNCTIONAL  CONTEXT,  OR  OTHER  MIXED 
APPROACH  WHERE  TRAINING  RESPONSIBILITY  MAY  SHIFT  BACK  AND  FORTH 
BETWEEN  THE  MILITARY  AND  CONTRACTOR?  (A  COROLLARY  ISSUE  IS  "HOW 
DOES  THE  FRONT-END  LOADING  OF  CONTRACTOR  TRAINING  RESTRICT  THE 
DESIGN  OF  THE  TOTAL  TRAINING  SYSTEM?") 

7.  WHAT  TESTS/MEASURES  ARE  USED  TO  INSURE  THAT  ALL  CONTRACTOR- 
SUPPORTED  TRAINING  OBJECTIVES  HAVE  BEEN  MET  (E.G.,  PAPER  AND  PENCIL 
TESTS,  PART-TASK  TRAINERS,  SIMULATORS,  AIRCRAFT,  ETC.)?  HOW  DOES 
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MILITARY  TRACK  THE  CONTRACT  PORTION  OF  TRAINING  TO  VERIFY  THAT  ALL 
OBJECTIVES  HAVE  BEEN  SATISFIED? 

8.  WHAT  ARE  THE  REQUIREMENTS  FOR  A  TRAINING  MANAGEMENT  INFORMATION 
DATA  BASE?  WHAT  KINDS  OF  DATA  SHOULD  BE  INCLUDED  AND  SHOULD  THERE 
BE  ONE  INTEGRATED  DATA  BASE  OR  SEVERAL?  ARE  REQUIREMENTS  DIFFERENT 
AS  A  FUNCTION  OF  THE  LEVEL  OF  CONTRACTOR  INVOLVEMENT?  WHAT 
PROBLEMS  ARE  INTRODUCED  BY  CLASSIFIED  TRAINING  INFORMATION  TO  WHICH 
THE  CONTRACTOR  DOES  NOT  HAVE  ACCESS? 

9.  WHAT  ARE  THE  GUIDELINES  FOR  THE  TEST  AND  EVALUATION  OF 
CONTRACTOR-SUPPORTED  TRAINING?  IN  THOSE  INSTANCES  WHERE  AN 
EXISTING  MILITARY  TRAINING  SYSTEM  (OR  FUNCTION)  IS  REDESIGNED 
AND/OR  TRANSITIONED  TO  A  CONTRACTOR,  HOW  IS  THE  COST-EFFECTIVENESS 
(OR  PAYOFF)  DETERMINED?  WHAT  ARE  THE  ADVANTAGES/ DISADVANTAGES  OF 
MILITARY,  CONTRACTOR,  AND/OR  THIRD  PARTY  EVALUATORS? 

10.  WHAT  MECHANISMS  CAN  BE  BUILT  INTO  A  SHARED  MILITARY/ CONTRACTOR- 
SUPPORTED  TRAINING  SYSTEM  TO  INSURE  THAT  IT  HAS  THE  CAPABILITY  TO 
INTEGRATE  CHANGES  IN  TACTICS,  DOCTRINE,  NEW  TECHNOLOGY,  ETC. 
EFFECTIVELY  AND  EFFICIENTLY?  FOR  EXAMPLE,  HOW  ARE  NEW  OR  DIFFERENT 
OPERATIONAL  REQUIREMENTS  DETERMINED?  HOW  DO  YOU  HEDGE 
UNCERTAINTIES/RISK? 

11.  WHAT  ARE  SOME  OF  THE  MAJOR  LESSONS  LEARNED  IN  THE  DESIGN, 
DEVELOPMENT,  EVALUATION,  AND  OPERATION  AND  MAINTENANCE  OF 
CONTRACTED  INSTRUCTION/TRAINING  WHICH  MAY  BE  APPLIED  TO  NEW  OR 
DIFFERENT  AIRCREW  TRAINING  PROGRAMS? 


32 


TITLE:  Aircrew  Training  System  (ATS)  Design  and  "Lessons  Learned" 
Database 

R&D  NEED:  Air  Force  experience  with  respect  to  the  design, 
development,  implementation,  evaluation  and  operation  of 
contractor-supported  aircrew  training  systems  is  extremely  limited. 
To  date  most  of  the  Air  Force's  major  ATS  programs  are  in  the  early 
stages  of  procurement  or  development.  As  a  result,  there  is  a  very 
limited  amount  of  empirical  and/or  experiential  information  to 
provide  guidance  during  the  acquisition,  development, 
implementation,  evaluation,  and  operation  of  new  or  proposed 
systems.  In  fact,  in  virtually  all  of  the  systems  reviewed  to 
date,  most  of  the  program  managers  began  at  ground  zero  and  had  to 
"learn  by  doing."  In  the  case  of  the  B-52/KC-135  Modernisation 
Program,  the  SAC  program  manager  is  benefiting  from  the  previous 
pre-contractual  experiences  of  AFHRL  and  MAC  with  the  C-130  ATS  and 
the  C-130  Model  Aircrew  Training  System  study.  However,  even  in 
these  instances,  the  experiences  shared  cover  only  the  very  early 
phases  of  system  conceptualization,  analysis,  specification,  and 
development.  In  any  case,  to  be  solely  dependent  on  occasional 
personal  contacts  or  undocumented  institutional  memory  for  guidance 
is  an  unacceptably  inefficient  procedure  for  the  development  of 
major  training  programs.  Thus,  there  is  a  critical  need  to 
collect,  store,  document,  and  disseminate  experientially  based 
lessons  learned  to  training  system  acquisition  offices  and 
operational  training  organizations  to  insure  a  cost-effective 
approach  in  the  design,  development,  and  utilization  of  total 
aircrew  training  systems.  Because  of  the  number  of  aircrew 
training  systems  at  various  stages  of  development  at  the  present 
time,  a  window  of  opportunity  is  available  for  initiating  a  study 
of  lessons  learned  during  various  phases  of  the  training  system 
life  cycle. 

PLANNED  R&D:  The  objective  of  this  effort  is  to  collect,  store,  and 
document  relevant  lessons  learned  by  Air  Force  and  contractor 
personnel  throughout  the  total  training  system  life  cycle. 
Particular  emphasis  will  be  devoted  to  lessons  learned  in  any 
aspect  of  system  design,  development,  implementation,  and  operation 
which  can  provide  useful  guidance  to  other  existing  or  proposed 
systems . 

The  approach  for  this  effort  consists  of  three  overlapping 
phases. 

Phase  1.  The  first  phase  includes  a  series  of  visits  tc  sites 
where  total  training  systems  are  being  planned,  developed,  c.  1  r’/cr 
operated.  Major  training  system  issues  and  lessons  learned  are 
being  solicited  from  both  contractor  and  military  personnel 
involved  in  all  phases  of  system  design,  development, 
implementation,  operation,  and  maintenance.  Much  of  the  initial 
information  is  being  developed  using  face-to-face  personal 
interviews.  During  these  initial  contacts,  specific  problem  areas 
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and  issues  are  being  identified  during  somewhat  open-ended 
discussions.  Where  feasible,  an  effort  is  made  to  obtain 
additional  information  concerning  what  problems  were  experienced 
and  what  actions  were  taken  to  resolve  them;  what  non-routine 
decisions  were  made  that  had  unexpectedly  good  (or  bad)  outcomes; 
and  what  approaches,  techniques,  and  procedures  were  developed  that 
might  be  of  value  to  others.  After  a  stable  set  of  potentially 
profitable  areas  and  issues  have  been  identified,  they  will  be 
monitored  through  periodic  visits,  telephone  communications,  and 
structured  questionnaires.  In  addition,  for  selected  high  priority 
areas,  key  program  personnel  will  be  requested  to  document 
important  activities,  decisions,  and  lessons  learned  in  special 
logs  or  journals. 

Phase  2.  During  the  second  phase,  a  computerized  database  will 
be  developed  and  maintained  for  the  storage  and  retrieval  of 
lessons  learned,  as  well  othei  relevant  training  system  design 
information.  This  database  will  be  the  initial  repository  for  the 
information  gathered  as  a  part  of  Phase  1  of  this  task,  as  well  as 
the  data  obtained  from  the  other  tasks  in  this  program.  In  its 
initial  configuration,  the  database  structure  will  be  designed 
using  a  relatively  limited  number  of  fields  reflecting  such  things 
as  the  type  of  system,  the  particular  stage  of  the  training  system 
life  cycle,  the  level  of  contractor  involvement,  the  instructional 
system  functions  and  training  resources  involved,  and  R&D  needs. 
As  more  information  becomes  available,  the  database  structure  will 
be  refined  to  increase  its  utility  for  both  training  system  design 
and  R&D  applications.  The  initial  database  is  being  implemented  on 
a  standard  Air  Force  Z-248  computer  using  DBase  III  Plus  for 
database  management.  This  should  provide  a  high  degree  of  access 
and  transportability  of  database  information  to  other  agencies 
and/or  locations. 

Phase  3.  During  Phase  3,  the  information  concerning  major 
issues  and  lessons  learned  will  be  documented  in  a  series  of  user- 
oriented  reports  and  guidelines  for  dissemination  to  procurement 
personnel,  training  system  designers,  training  system  acquisition 
agencies,  training  managers,  and  R&D  personnel.  The  guidelines 
should  effect  a  significant  improvement  in  the  effectiveness  and 
efficiency  of  the  system  design,  development  and  utilization 
processes.  In  addition,  the  computerized  database  should  be  a 
prime  source  of  guidance  for  identifying,  organizing,  and 
prioritizing  significant  training  system  R&D  issues. 
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The  following  is  a  brief  description  of  the  contractor- 
supported  aircrew  training  program  for  the  U.S.  Air  Force's  E-3A 
system.  It  is  provided  to  serve  as  an  example  of  the  level  of 
detail  desired  at  this  time  for  the  descriptions  of  contracted 
instruction/training  submitted  with  your  issues  and  lessons  learned 
inputs . 

TITLE:  CONTRACT-SUPPORTED  TRAINING  FOR  THE  E-3A  WEAPON  SYSTEM. 

The  Tactical  Air  Command  of  the  U.S.  Air  Force  awarded  a 
contract  to  the  Boeing  Military  Aircraft  Company  in  April  1984  to 
provide  E-3A  aircrew  initial  qualification  training.  The 
contractor  designed  the  training  program,  purchased  a  computer- 
based  training  system,  developed  courseware,  and  acquired  two  707 
aircraft  for  use  as  surrogate  E-3A  trainers.  Contract  personnel 
conduct  all  ground  training  (academics,  simulator,  part-task 
trainers,  etc.)  for  the  initial  qualification  of  E-3A  pilots, 
navigators,  and  flight  engineers.  The  Air  Force  conducts  all 
formal  simulator  and  in-flight  evaluations  as  well  as  all  in-flight 
training  of  pilots  and  copilots  in  the  surrogate  aircraft.  During 
these  training  flights  contractor  personnel  serve  as  flight 
engineers.  The  contractor  is  also  responsible  for  the  operation 
and  maintenance  of  all  instructional  resources  including  the  707 
aircraft,  flight  simulators,  and  the  computer-based  training 
system.  The  benefits  realized  from  this  program  include:  (a) 
reduction  in  the  use  of  very  expensive  E-3A  airframes  for 
proficiency  training  (this  has  led  to  an  increase  in  the  number  of 
operational  missions  supported  by  the  available  E-3A  fleet) ;  (b)  a 
more  stable  contractor  instructor  force;  and,  (c)  a  significant 
reduction  in  the  costs  associated  with  initial  qualification 
training.  (Estimated  cost  prior  to  contract — $22M/year.  Estimated 
current  costs — $10-12M/year . ) 
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APPENDIX  B:  ADDITIONAL  "LESSONS  LEARNED"  DATA 
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SYSTEM  DESCRIPTION 

The  Tactical  Air  Command  of  the  U.S.  Air  Force  awarded  a 
contract  to  the  Boeing  Military  Aircraft  Company  in  A^ril  1984  to 
provide  E-3A  aircrew  initial  qualification  training.  The 
contractor  designed  the  training  program,  purchased  a  computer- 
based  training  system,  developed  courseware,  and  acquired  two  707 
aircraft  for  use  as  surrogate  E-3A  trainers.  Contract  personnel 
conduct  all  ground  training  (academics,  simulator,  part-task 
trainers,  etc.)  for  the  initial  qualification  of  E-3A  pilots, 
navigators,  and  flight  engineers.  The  Air  Force  conducts  all 
formal  simulator  and  in-flight  evaluations,  as  well  as  all  in¬ 
flight  training  of  pilots  and  copilots  in  the  surrogate  aircraft. 
During  these  training  flights,  contractor  personnel  serve  as  flight 
engineers.  The  contractor  is  also  responsible  for  the  operation 
and  maintenance  of  all  instructional  resources  including  the  707 
aircraft,  flight  simulators,  and  the  computer-based  training 
system.  The  benefits  realized  from  this  program  include:  (a)  a 
reduction  in  the  use  of  very  expensive  E-3A  airframes  for 
proficiency  training  (this  has  led  to  an  increase  in  the  number  of 
operational  missions  supported  by  the  available  E-3A  fleet) ;  (b)  a 
more  stable  contractor  instructor  force;  and  (c)  a  significant 
reduction  in  the  costs  associated  with  initial  qualification 
training.  (Estimated  cost  prior  to  contract:  $22M/year.  Estimated 
current  costs:  $10-12M/year . ) 


E-3A  ISSUES/LESSONS  LEARNED 

1.  PROGRAM  ADVOCACY:  Involve  senior  Air  Force  leadership  early  and 
get  their  names  on  the  dotted  line. 

2.  TRAINING  EMPHASIS:  Address  each  crew  position  with  sufficient 
emphasis.  Navigator  training  appears  to  be  underemphasized  in 
E-3  program. 

3.  MANAGEMENT:  Retain  management  reserve  funds  in  contract  to 

handle  contingencies. 

4.  MANAGEMENT:  Provide  a  USAF  liaison  at  each  of  the  operational 
units. 

5.  INFORMATION:  Engage  in  crosstalk  with  other  programs  to  profit 
from  experiences  of  other  systems. 
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